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ABSTRACT 

Literacy policy has often developed independently of 
other social and employment programs. As a consequence, literacy 
tends to become an end unto itself, and assessment is directed more 
toward academic, archival ends than toward policy evaluation. Many 
justifications given for large-scale literacy programs are not based 
upon empirical data. Foremost among these jus t if icat i ons are the 
projected skills mismatch between the job market and new entrants to 
the work force. Additionally, the target audience for literacy 
training is often greatly inflated. The basis of most recent 
assessments of adult literacy is functional literacy, but there are 
concerns that it fails to provide a basis for predicting abilities in 
new literacy contexts and it lacks a basis for selecting specific 
assessment tasks. Rather than abandoning functional literacy, an 
attempt should be made to support it both theoretically and 
empirically. Questions have also been raised about the effectiveness 
of different literacy programs and limited experience in measuring 
literacy program performance. The need for an extensive database to 
support literacy policy becomes imperative. Perhaps the most 
important lesson to be learned from national literacy surveys is that 
they have limitations for policy formation and evaluation. What is 
needed m most developed countries is more focused information on the 
groups who need assistance and the effectiveness o programs 
instituted to assist them. (Contains 41 references.) (YLB) 
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LITERACY ASSESSMENT 
IN THE SERVICE OF 
LITERACY POLICY 



Richard L. Venezky 
University of Delaware 



Abstract 

Literacy policy has often developed independently of other social and 
employment programs. As a consequence, literacy tends to become an end unto 
itself, and assessment is directed more toward academic, archival ends than 
toward policy evaluation. Although the need for improved literacy training for 
individuals and identifiable subpopulations in the United States is well 
established, many of the justifications given for large-scale literacy programs 
are not based upon empirical data. Foremost among these justifications are the 
projected skills mismatch between the job market and new entrants to the 
workforce. Additionally, the target audience for literacy training is often greatly 
inflated, usually by ignoring the age distribution of out-of-school persons who 
lack a high school diploma or equivalent. New assessment procedures should 
be built around the long-term objectives of literacy training — economic 
independence, social integration, and personal satisfaction. These procedures 
should also provide information for evaluating different policy options. 
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Introduction 



Literacy assessments like the Canadian Survey of Literacy Skills Used in 
Daily Activities (LSUDA), the U.S. National Adult Literacy Survey (NALS). 
or the just completed International Adult Literacy Survey (lALS; OECD, 1995) 
should be niotivated by clear sets of policy objectives. These objectives, in turn, 
should derive from repeated cycles of information gathering and policy 
formation, with refinement of methods and convergence of objectives in each 
cycle, leading systematically to a focused national or international survey. In the 
real world of unemployment, AIDS, and cabinet shakeups, however, literacy 
policies often become adjuncts to other social, political, and educational 
interests, and the information base for literacy policy formation becomes weak, 
outdated, and cursorily analyzed. It is, perhaps, a compliment to the 
policymakers of the major industrial countries that so much has been 
accomplished with adult education in the past decade in spite of severely limited 
knowledge bases in most countries and equally impoverished analysis 
procedures. Nevertheless, a chasm continues to exist between literacy policy 
and literacy assessment in those industrialized countries that have produced 
evidence of both. It is the goal of this report to explore this gulf, particularly in 
the United States, but with the intention of proposing how it might be avoided 
or closed in other countries. 



Policy Justification 



Literacy Las been an interest of the federal government in the United States 
at least since the 1880s when Senator Henry W. Blair of New Hampshire 
proposed federal aid to education based upon the number of illiterates in each 
state who were over the age of 10. More recently the Adult Education Act (P L 
100-297), inter alia, provided federal funds to the states for adult basic 
education (ABE), adult secondary education (ASE), and English as a second 
language (ESL). President Bush, in America 2000, and President Clinton, in 
Goals 2000, targeted adult literacy as a national educational goal. In recent 
legislation, as well as in earlier debates, literacy has been offered as a proxy for 
basic education, including communicative competence in English for those who 
are not native speakers of English. Although this report will continue to 
emphasize literacy in conjunction with policy formation, it is important to 
interpret this term in a broad sense, covering all that is incorporated in adult 
basic education today. Literacy in a strict sense is inadequate to accommodate 
the breadth of personal, social, and economic goals expressed in the 
justifications for federal adult literacy programs. Definitions of literacy adopted 
by programs and surveys, because of this disjunction, often fall into rhetorical 
I 990 T' components that are seldom taught or assessed (Venezky, 

The logic of federal literacy policy derives first from Constitutional 
prerogatives that center on the health and welfare of individual citizens, and by 
extension reach social and economic concerns of the nation* itself 
Policymakers, through a variety of different types of evidence, have been 
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convinced that a significant number of citizens lack basic education and that 
as a result of this deficiency, various problems of national concern exist, 
centered on equity, security, and economic competitiveness. Foremost 
among these are the following: 

• Intergenerational poverty. Many families, including a 
disproportionate number of minorities, have experienced 
intergenerational poverty and have little hope under their present 
circumstances of achieving economic independence. These 
families tend to be dependent upon the welfare system for basic 
necessities, to have children out of wedlock, particularly at ages 
when the mothers should be completing their schooling, and to 
have limited participation in civic and community activities. 

• Integration of immigrants. Related to intergenerational poverty is 
the plight of immigrants who do not spe^ English sufficiently 
well to communicate in daily life and in the workplace. The 
integration of these people into the mainstream of American life 
requires that they leam to speak English well and that they acquire 
basic skills if they did not have them when they came to this 
country. 

• Recidivism of released criminals. Basic skills among inmates of 
correctional institutions tend to be significantly lower than those of 
the general population, and that fact is often cited as a cause of 
recidivism (Ross, 1990; Ross & Gendreau, 1980). The current 
emphasis on educational programs, particularly literacy programs, 
in correctional institutions is driven in part by data that suggest a 
lower recidivism rate for those with higher educational attainment 
(Newman, Lewio, & Beverstock, 1993). Related to these facts, 
there is a widespread belief that crime rates will be reduced if 
educational levels can be raised, particularly among interurban 
poor. 

• Workforce development. A number of reports over the past decade 
have projected a rapid decline in low skill jobs and an equally rapid 
increase in the skill requirements of existing and projected jobs 
(e.g., Johnston & Packer, 1987). These reports, coupled with 
uncertainty over the quality of the U.S. educational system (e.g.. 
National Commission, 1983), have created a concern for the 
ability of the U.S. workers to contribute significantly to America’s 
global economic competitiveness. 

Although other basic justifications for federal literacy policy might be 
adduced, including the strengthening of democratic ideals, the dignity of 
individual citizens, and the survival of political institutions, the issues 
outlined above are the ones most often cited as justifications for literacy 
funding and are also defined at a level of specificity that can easily be 
translated into both survey and policy needs. 

It is important to note that these four factors are highly interdependent. 
Without marketable skills, including English language proficiency, poverty is 
a probable outcome. For the incarcerated, as well as for immigrants and 
native poor, the route to economic independence is through gainful 
employment, which requires both entry-level skills and then acquired 
competence. Literacy, even in the broad sense, is an enabling skill. By itself 
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it has little market value. It is, like a password, an entree to the types of job 
training that can lead to sustainable employment. What is left unexplored in this 
report is the importance of the country’s and each region’s economic well-being 
for the integration of the disadvantaged into the workforce. Although the 
interest here is literacy policy and assessment, the importance of other 
educational, social, and economic variables in moving individuals and families 
into the mainstream of American life should not be ignored. The remainder of 
this report will be an exploration of the key policy issues raised by these 
concerns and the types of data that are needed to ^ve coherent literacy policies. 



Key Policy Issues 



The Skills Mismatch Debate 

One of the most important policy issues centers on workforce needs. Recent 
reports have attempted to establish a “labor shortage” or “skills mismatch” 
perspective of labor market trends. According to this view, the rapid movement 
away from manufacturing jobs and toward service jobs in the United States is 
leading and will continue to lead to an equally rapid growth of technical and 
professional jobs. However, labor force expansion will be dominated by low 
skill minority workers, thus resulting in a skills mismatch between jobs and 
available workers (Johnston & Packer, 1987; Mishel & Teixeira, 1991). 

At least one major study has challenged this view, projecting that 
occupational upgrading will slow dramatically in the near future, and that only 
about one third of the entrants to the labor force in the 1990s will be minorities 
(Mishel & Teixeira, 1991). According to this analysis, non-Hispanic Whites 
will comprise two thirds of the entrants. Furthermore, wages for most workers 
without college degrees fell during the 1980s and remain below their late 1970s 
levels. Newer jobs with higher skill requirements are not paying wages 
sufficiently above those jobs with lower skills to rectify this decline in real 
wages. This report suggests an emphasis on the restructuring of work to 
improve production system performance, with major attention given to 
improving the “jobs, pay, and skills of the non college-educated workforce” (p. 



Other data published recently show that productivity in the United States is 
the highest in the world and is increasing at a faster rate than that of any other 

example, the average American worker produced 
M9,600 in goods and services; this was 1 1.2% higher than the output of the 
average German worker and 22.4% higher than that of the average Japanese 
u 1991 to 1992, factory productivity in the United States increased 

by 4.3% while that of Germany increased by only 0.5% and that of Japan fell 
by 5.0%. In the service areas America’s lead in productivity over Japan and 
Gemiany is even wider (Nasar, 1994). Due to these (and other) changes, the 
United States’ share of the world export market is now increasing after many 
years of decline. Economists project, furthermore, that these trends will not 
change in the near future, due to reductions already made in costs in both 
manufacturing and in the service industries. 
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Quality of American education 

Related to the skills mismatch debate is an assumption, fueled by mis- 
interpretation of NALS outcomes, that large numbers of high school and 
college graduates have inferior literacy skills. A Nation at Risk^ issued in 
1983, gave the impression that American education was in a state of crisis, 
that enormous numbers of students were dropping out of school, and that 
those who remained until graduation seldom acquired the skills needed to 
survive economically (National Commission, 1983). More recent data either 
refute or cast considerable doubt on these conclusions. For example, a study 
by the Sandia Laboratories showed that reported declines in SAT scores were 
due to a changing mix of students taking these examinations and not to a real 
reduction in abilities of high school students (Carson, Huelskamp, & 
Woodall, 1991). More recently, an OECD report has found the American 
educational system to be among the healthiest in the world (Cells, 1993). 
Students in the United States score above average in reading comprehension 
and about average in science. The percentage of the population between the 
ages of 25 and 64 who hold college degrees (36%) is among the highest in 
the world and the percentage of women enrolled in colleges and universities 
is the highest among all OECD countries. 

These studies do not invalidate the need for improved literacy training for 
some high school and college graduates, nor do they dilute the need to 
provide training for new immigrants who lack English language or other 
basic skills, but they do call into question claims that vast numbers of U.S. 
high school and college graduates are in need of literacy training. For 
policymakers, there is a need to go beyond the gross statistics and simple 
correlational data to determine who obtains adequate literacy skills from high 
school or college education and who does not. It is equally important to 
determine which high schools and colleges are imparting adequate literacy 
skills and which are not. Some of these data can be extracted from the NALS 
database, although earnings data are somewhat crude. 

Target Audiences 

If we accept that some people, regardless of number or cause, need 
further literacy training, then the identification of this target audience for 
literacy programs becomes an issue. The most prevalent identification 
technique is to target persons 16 years of age and older who are out of school 
and still without a high school degree or equivalent. According to the 1990 
household census, about 23.7% of all persons out of school and 16 years of 
age or older, or 45.4 million people, could be placed in this category (Thome 
& Fleenor, 1993). 

However, this figure inflates the expected demand for literacy instruction 
because it ignores age and economic circumstances. Among 16- to 24-year- 
olds, for example, only 11% are out of school and without a high school 
degree or equivalent while in the 50 and above age level, many more than 
24% are without a high school diploma or equivalent. That this older group 
is not seeking further educational certification can be seen from the GED 
testing data that show that for 1991, only 8% (64,483 persons) of those who 
attempted the GED Tests were 40-49 years of age and only 2.8% (22,569) 
were 50 or older (GED Testing Service, 1992). We do not know how many 
people in the younger age ranges who are out of school and without a high 
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school degree are making a satisfactory income, but we suspect that it is not an 
insignificant number. 

Partially counterbalancing these figures are the numbers of individuals with 
high school and post-secondary degrees who have inadequate literacy skills and 
who seek further training. According to National Evaluation of Adult Education 
Programs (NEAEP) Survey, 33% of those who attended ESL, ABE, and ASE 
classes between mid-April 1991 and mid-April 1992 had a high school diploma 
or post-seconda^ degree (Development Associates, Inc., 1993). (However, if 
the population is restricted to ABE and ASE students, this figure reduces to 
11 %.) 

It is doubtful that any single criterion will completely describe the target 
audience for literacy programs, given the variation that normally occurs in 
individual aspirations and in labor skill needs. Basing all of the need upon 
degree attainment is inadequate for the reasons just given. Similarly, literacy test 
performance is also an inadequate indicator, not only because of individual and 
occupation variations, but also because of the complexity of linking literacy test 
performance to workplace or daily life functioning. This problem is illustrated 
in Table 1, which shows reported amount of job-related reading of reports, 
articles, magazines, or journals by persons working in Construction crafts. This 
occupation is 1 of 40 derived from responses to an occupation question on the 
NALS background questionnaire; data in each cell are based upon population 
projections for persons 25 years of age or older who were bom in the United 
States or its territories. 



Table 1 

Percentage and document literacy scale score of projected employees of 
the construction crafts for reading or using Information from reports, 
articles, magazines, or journals 
(N[projected]= 4,431,735) 



Frequency of Reading 





Everyday 


A few times 
a week 


Once a week 


Less than 
once a 
week 


Never 


Percentage 


13.6 


17.2 


11.9 


25.2 


32.3 


of 

employees 
Mean (s.d.) 


274 


282 


275 


284 


252 


document 

literacy 


(48.9) 


(45.3) 


(59.9) 


(53.2) 


(62.9) 



Note. The actual number surveyed for each frequency of reading was 74 (Everyday), 90, 
64, 137, and 156 (Never). 



What is especially striking is that while almost 31% of the projected 
workers claimed to read reports, articles, magazines, or journals daily or several 
times each week, over 32% claimed that they never engage in this type of 
reading on the job. The distribution cf 37% of the projected population across 
the other two response categories shows that the respondents cannot be divided 
simply into managerial and manual categories. Furthermore, there is no 
consistent relationship between mean literacy performance and frequency of job 
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reading. What is represented here is probably not only a wide diversity of job 
types under a single definition, but dso a large variation within certain jobs, 
no matter how narrowly defined, in amount of reading required or elected. 
Although minimal levels of literacy ability might be established for some jobs 
like these, a vast, foggy middle range of literacy requirements will remain for 
the majority of the available occupations because literacy is only one of many 
skills needed for most employment opportunities. 

The challenge for policymakers is to differentiate target audiences 
according to needs and life circumstances so that appropriate programs can 
be identified. High school dropouts who are still in their teens require a 
different instructional program than that required by 30- and 40-year-olds 
who are seeking GED certification or improved skill training for 
employment. Non-English speaking immigrants who arrive in the United 
States with minimal education from their native countries will need far more 
help to become self-sufficient than those who arrive with college or 
postgraduate degrees. Workers who seek further skills for job advmcement 
require different programs than those who are unemployed and without the 
basic skills required for entiy-level jobs. And those with learning disabilities 
require more extensive assistance than those who are not confronted with 
such problems. A bottom level might be established on a literacy assessment 
for identifying those who need further help regardless of their aspirations or 
life situations. Similarly, assessments of English language competence might 
identify those who need further English-language training for ordinary 
citizenship and for entry-level employment. Beyond these identifiers, 
however, other data will be needed for identification of specific needs. 



A consideration of the type of literacy that ought to be assessed rarely 
enters the policy debate. The basis of NALS as well as most other recent 
assessments of adult literacy (e.g., APL, Louis Harris and Associates, 
YALS) was functional literacy. Although this term was used by the Civilian 
Conservation Corps in the 1930s in reference to persons with less than three 
years of education, /Mncrio/m/ literacy as used in literacy assessments today 
appears to derive from UNESCO usage in the 1950s (Gray, 1956; 
UNESCO, 1957). William S. Gray, a leading American educator of the time 
and America’s foremost expert in reading development, authored one of the 
UNESCO works on worldwide literacy and defined functional literacy in 
terms of the reading and writing skills needed for functioning in everyday 
life. Gray defined a functionally literate person as one who possessed the 
“knowledge and skills in reading and writing... to engage effectively in all 
those activities in which literacy is normally assumed in his culture or group” 
(Gray, 1956, p. 24). 

The UNESCO/Gray definition, although circular, has been the basis of 
almost all definitions of functional literacy since the 1950s, but subtle 
differences have occurred across these formulations. For example, the U.S. 
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Right -to-Read Program in the 1970s declared that a literate person should be 
able to “read and understand the whole spectrum of printed materials that one is 
likely to encounter in daily life,” thus eliminating writing from the definition 
(Carroll & Chall, 1975, p. 8). Statistics Canada, although not employing the 
itxm functional literacy, implied such in arguing for an empirical and operational 
definition of literacy. Their definition (1991), “The information processing 
skills necessary to use the printed material commonly encountered at work, at 
home, and in the community,” also leaves out writing but, more importantly, 
stresses information processing skills (Neice & Adsett, 1991, p. 3). 

The YALS and NALS definition, “Using printed and written information to 
function in society, to achieve one’s goals, and to develop one’s knowledge and 
potential,” goes beyond simply functioning in society (Kirsch, Jungeblut, 
Jenkins, & Kolstad, 1993, p. 2). Deciding what abilities are required to use 
printed and written materials to function in society is a difficult task and one that 
has yet to be examined fully. Whether this decision assumes a single adequate 
level of functioning in society or different levels for different subpopulations, 
and how this level (or levels) is determined are just two of the questions left 
unanswered by this definition. Add onto this challenge that of deciding what is 
required to achieve one’s goals and to develop one’s knowledge and potential, 
and one is faced with a Herculean labor beyond that of cleaning the Augean 
Stables or driving off the Stymphalian "Birds. However, since neither YALS nor 
NALS paid any attention to these components of their definitions, no further 
attention will be given to them here. From a policy perspective, however, the 
vagueness in the definition of functional literacy renders evaluation Of functional 
literacy programs nearly impossible. 

Functional Versus Other Forms of Literacy 

The expected contrast for functional literacy is not non-functional literacy 
but academic literacy (i.e., the skills and abilities taught in the school literacy 
programs). These are reflected in the various NAEP reading and writing 
assessments, which tend to emphasize skills like critical comprehension and 
reference skills. But the contrast between functional literacy and most other 
forms of literacy is not between differing views of what literacy is but between 
differing interests in basic and applied abilities. Tests like the NAEP Reading 
and Writing Assessments, as well as school-based tests like the CAT and the 
CTBS, attempt to evaluate the information processing skills that underlie 
ordinary reading and writing tasks. These tests ask first, “What do we kuow or 
assume about the cognitive processing of texts?” In contrast, NALS asks, 
“What texts do people use in everyday life and what do they need to do with 
them?” 

Assessments of basic skills can be translated directly into instructional 
policy; in addition, they can, at least in theory, be used to predict abilities in 
applied situations. In contrast, functional literacy assessments have no direct 
instructional implications. What should teachers do, for example, when a high 
percentage of the adult population scores low on a task requiring the reading of 
restaurant menus? Introduce more exposure to menus into the curriculum? This 
may have some logical support but not very much. Extended to its logical 
conclusion, it would result in a language arts curriculum composed of 12 
minutes per day of work on menus, 8 on bank deposit forms, 7 on parts 
catalogues, 5 on bus schedules, and so on. In Gulliver’s Academy of Lagado, 
this might be desirable, but not in Western education. Even the most severe 
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critics of American schooling agree that the schools should teach general 
skills, not vocational applications. The school language arts curriculum 
desperately needs more emphasis on everyday documents and expository 
writing, but in the context of critical thinking and problem-solving skills. 

A second concern with the functional literacy approach is that it fails to 
provide a basis for predicting abilities in new literacy contexts. Imagine, for 
example, that, in five years, literacy demands change dramatically and new 
types of documents become common. Could we predict how well adults 
might do on these documents based upon their performance on present-day 
functional literacy tests? Possibly yes and possibly no, depending upon the 
processing skills shared by the tested individuals and upon new tasks and 
documents. Since a functional lite;:acy assessment is not developed from a 
‘ model of underlying skills, we have no guarantee that any specific skill will 
be assessed. The skills required by NALS items, for example, are a function 
of the specific documents and tasks selected. No model or even list of 
processing skills was created as a basis for item selection. By this method, 
minor skills may be over represented and important ones ignored. 

A third concern is the basis for selecting specific assessment tasks (i.e., 
the knowledge base for deciding what texts and tasks characterize 
functioning in society). Although some work has been done on this question, 
far more work is needed to build a comprehensive survey. If we take YALS 
and NALS as representations of functional literacy, then a number of 
problems are immediately apparent. First, in real life, people need to read and 
understand lengthy texts: a chapter on nutrition for infants, instructions for 
selecting health insurance coverage, options for college majors, and so on. 
The longest text to be read in NALS falls far short of these. 

In addition, many reading tasks need to be done in a brief amount of time 
(e.g., reading a flight schedule on the run at an airport to find an alternative 
flight to the one that was just canceled or evaluating a memorandum handed 
to you in the course of a working meeting). Many low literate adults can 
puzzle out such documents given long amounts of time; however, if pressed 
to do them quickly, they may fail. Studies of reading processes and of 
reading disabilities have found a consistent pattern of processing speed 
deficits in disabled and low functioning readers (Bowers & Swanson, 1991; 
Stanovich, 1986; Wolf, 1991). But speed of processing is not tapped by 
NALS nor by any other present-day adult literacy test. 

Beyond these processing issues are matters of text selectivity. For every 
person in America who reads Emily Dickinson, probably 100 read the 
National Enquirer and 1000 read the L.L. Bean and J. Crew catalogues. Yet 
Emily Dickinson’s poetry is part of NALS and the National Enquirer and the 
clothes catalogues are not. The NALS also omits lottery tickets, dismissal 
slips, welfare notices, credit collection demands, and warrants for one’s 
arrest. NALS texts represent a sanitized, upright, and respectful lifestyle — 
not the underbelly of America that many in the population have to cope with 
everyday. And the reading habits of those who own second homes at Lake 
Tahoe, buy toys for their children and grandchildren at F.A.O. Schwartz, 
and munch on Godiva chocolates are as underrepresented in NALS as are 
those of people who collect welfare checks each month. There are no items 
built around limited partnership brochures, Mercedes Benz checkup notices, 
or American Express Gold Card applications. If a national survey is to tap 
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the entire spectrum of literacy abilities in America, then the population needs to 
be stratified by living style and the print-based habits of each stratum examined. 

Toward a Theory of Functional literacy 

One resolution of the problems just described is to abandon functional 
literacy as a basis for national literacy surveys, replacing it with basic skills 
approaches. This would align adult literacy assessment with K-12 literacy 
assessment, and therefore provide a continual, lifelong measurement of literacy 
abilities. As desirable as this might appear in the abstract, it could have 
undesirable practical consequences. K-12 testing procedures for adults would 
reinforce K-12 instructional procedures at the adult level, thus continuing a 
practice that has been unproductive in the past. Adult learning can be 
distinguished from school-age learning in a variety of ways; however, what is 
most important is that the majority of the adults in tutoring, in ABE, ASE, and 
GED programs had unsuccessful experiences in elementary and secondary 
schools and do not respond well to instructional settings and practices that 
duplicate these. 

In addition, most adults in basic skills and literacy programs have everyday 
responsibilities that keep them focused on the here and now— jobs to find and 
maintain, children to raise, rent and utility bills to pay, and so on. Although 
intellectual enrichment is desired for everyone, coping with daily needs is a 
prior necessity for most adults. Functional literacy, by definition, provides a 
focus on practical matters associated with print. It has an ecological validity that 
is obvious to most who encounter it in assessments and in instruction. 

Therefore, rather than abandoning functional literacy, an attempt should be 
made to support it both theoretically and empirically. The first step in doing this 
has already been suggested — stratify the population by living style or whatever 
other variables might be relevant for classifying social functioning and then 
define the print-related habits of each stratum. This will produce a pool of texts 
and tasks for building an assessment. Most functional assessments since the 
1970s have done something approaching this but then have failed to analyze the 
resulting pool to understand the underlying demands of each text-task pair. 
Instead, surveys were built on matrix sampling of the resulting pool. A more 
adequate survey could be built if the processing demands of each text-task pair 
were defined within the framework of a specified cognitive processing model 
and these processing demands formed the basis of selection from the pool. In 
this way, everyday literacy tasks would still be used, but they would be 
grounded in a theory of cognitive processing that would guide interpretation of 
the results. Clusters of items would represent different cognitive processes, 
thus allowing reporting not only of abilities on specific tasks, as is done now,’ 
but also reporting of specific information processing skills. 

This approach should also lead to a reinterpretation of the scales for surveys 
like NALS. The present division into prose, document, and quantitative scales 
has little empirical or theoretical support. It is questionable, for example, 
whether the quantitative scale adds any value to the survey beyond that of the 
other two scales. It is not an assessment of adult quantitative abilities per se and 
therefore could not (and should not) be used for decisions about mathematics 
ability. With the importance of quantitative ability today for work and home life. 
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workers with solid basic skills rather than narrow, functional training. 
Somehow the best of both the functional and the basic skills worlds needs to be 
taught to under-skilled adults. Perhaps a principled functional literacy approach 
can do this. 



The Effectiveness Assumption 



An assumption of most literacy policies is that if adults could only be 
persuaded to attend adult basic education programs, or other types of literacy 
instruction, they would acquire the skills they are presumed to lack. This 
assumption is relatively untested. We know little about the effectiveness of 
different literacy programs and have limited experience in measuring literacy 
program performance (Darkenwald, 1986). 

Most carefully controlled studies of program effectiveness done over the 
past five years have reported either ambiguous or negative findings. In one of 
the largest of such studies. Manpower Demonstration Research Corporation 
reported recently on the effectiveness of the New Chance Program, which 
enrolled 1,408 teenage mothers in 10 states in literacy, job training, and life 
skills classes (DeParle, 1994). After 18 months, program participants had the 
same welfare and employment rates as a matched control group that received no 
training or other social services. Only small increases in parenting and literacy 
skills were found, and about 57% of the women in the program got pregnant in 
the 18 month period, compared to 53% of the control group women. 

A study of 825 parents who received literacy instruction in Even Start 
programs reported small but educationally insignificant increases in literacy 
ability after 40 or more hours of instruction. Similarly, a study of 92 adults in 
ABE and ASE programs in New York State found contradictory evidence for 
reading gains after eight months of instruction. Changes in performance on 
functional literacy, reading comprehension, mathematics, and vocabulary 
measures showed a range of outcomes from a gain of about 0.4 standard 
deviations to a loss of about 0.25 standard deviations, depending upon the test 
used (Venezky, Bristow, & Sabatini, in press). 

Finally, the New York City data mentioned earlier showed a gain of 9.9 
months in reading ability for the first year of instruction, but only 3.3 months 
for the second year of instruction, and 3.0 for the third year, with a mean of 
212.4 contact hours per year (Literacy Assistance Center, 1992). At this rate, a 
person who enters a program reading at a beginning sixth-grade level can expect 
to advance only to a mid seventh-grade level after three years of course work. 

Although these averages do not reveal how many persons do make 
meaningful gains as a result of literacy instruction, the general lack of 
significant advances across programs has serious implications for 
policymakers. Should adults who are performing below a mid high school level 
be encouraged to take academically oriented basic skills programs? Is GED 
certification a reachable goal for a majority of the adults who enroll in literacy 
programs? 
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One assessment goal should be to identify the conditions under which 
adults make meaningful gains in literacy programs. At present, we have 
almost no good estimators of the progress adults will make in literacy 
programs. We do not know if the variance in literacy outcomes from 
instruction is due to quality of instruction, amount of out-of-class reading, 
attendance patterns, intelligence, or any other variable or set of variables. 

Almost all program output measures are of academic skills, yet most 
researchers agree that attitudes and beliefs also deserve attention. Even the 
roles of some types of programs are in doubt. Library tutoring programs, for 
example, usually provide services to adults at the lowest ability levels, using 
relatively untrained volunteers for tutors. Do we expect appreciable literacy 
skill gains from such programs or do we expect more confidence building 
and life skill acquisition? To what degree should these programs prepare and 
encourage adults to attend ABE classes? 

Another assumption is that adults who acquire whatever is being taught 
in literacy programs will be better equipped for high-skilled jobs in the labor 
market, will function better as parents and home managers, and will 
participate more fiilly in civic and community affairs. Once again data are too 
limited to draw strong conclusions; nevertheless, several recent reports raise 
doubts about the effectiveness of job training or retaining programs. The 
MDRC study described above documents one instance of a lack of positive 
outcomes (DeParle, 1994). A similar study, done internally by the 
Department of Labor, concerning government-sponsored job retraining 
programs, found even more dismal results. Of 1,198 workers in 9 states 
who received job retraining, only about 19% found jobs within a year of 
completing the program that drew upon their training and paid 80% or more 
of their previous wages. The Labor Department report on the study 
concluded that the job training program was “largely ineffective and used 
primarily as an income crutch for the unemployed” (Kilbom, 1993, p. Al). 

Further complicating evaluation of program effectiveness is the open 
entry/open exit policy incorporated into most literacy programs, with 
students defining their own goals. Although this policy may be successful in 
encouraging adults to attend programs where they might not be comfortable 
under different operating procedures, it does not ensure that adults will be 
striving for the levels of literacy that they need and could attain. Adults with 
low literacy levels tend to lack confidence in their own abilities to learn. 
Without assistance in defining what their needs might be, they may aim for 
far less than what is necessary for their own success. 

More serious is the lack of basic understanding of the literacy demands of 
home and civic functioning. We assume, for example, that one goal of 
literacy instruction is to improve parenting skills, particularly for fostering 
the literacy development of children, yet we have only crude estimates of the 
amounts and types of reading, writing, and mathematics that are required to 
succeed at this task. Like most of the other goals of literacy instruction, 
parenting is dependent upon much more than literacy. In some cases, literacy 
is only a proxy for some complex of skills and may not be necessary, given 
compensating conditions. For example, many immigrant and refugee parents 
in America who are not literate in English are nevertheless quite effective in 
fostering their children’s education in English-speaking classrooms (Caplan, 
Choy, & Whitmore, 1992). 
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The Role of Testing 



Given the questions raised about the justification for federal expenditures 
for literacy instruction, in defining the target audience for such funding, in 
deciding on what type of literacy to assess, and in demonstrating effectiveness 
of literacy programs, the need for an extensive database to support literacy 
policy becomes imperative. In recent years, strong steps have been taken in a 
few countries toward this goal. These have included not only national surveys 
of literacy performance (e.g., LSUDA, NALS), but also surveys of federally 
funded programs and their clients (Development Associates, Inc., 1993), 
analyses of decennial census data for literacy needs (Thome & Fleenor, 1993), 
and evaluations of specific literacy program (DeParle, 1994). Because of the 
progress made in these surveys and evaluations, we are in a better position now 
than ever before to reflect on the types of data collected and their role in policy 
formation. In this section, several issues are raised about specific types of 
literacy assessments. In the concluding section, further needs fcr literacy policy 
formation and evaluation are presented and recommendations for data gathering 
are made. 

Population Assessments 

National and state surveys of literacy involve two types of data collection: 
background information, generally gathered orally, on educational, economic, 
health, language, and civic activities and experiences; and skill abilities as 
measured by some nationally standardized instrument. As discussed above, 
tests of functional abilities have been favored since the early 1970s for 
population surveys of adult literacy. These tests tend to have a high face validity 
for adults because they draw their items from everyday literacy tasks that are 
familiar to most English-speaking adults. The most recent population surveys, 
the Young Adult Literacy Survey (YALS; Kirsch & Jungeblut, 1986), the 
National Adult Literacy Survey (NALS), and the Canadian National Survey 
(LSUDA), were constructed around short answer responses to everyday 
reading tasks. Although some arithmetic ability is required in one set of tasks 
and some responses require writing, no systematic attempt is made to assess 
these skill domains. In addition, both skills are so embedded within reading 
contexts that it is not always possible to determine where an item difficulty 
rests. 

In contrast, national surveys of elementary and secondary level literacy 
abilities are based upon basic reading, writing, and mathematical skill definition 
(e.g., the National Assessment of Educational Progress or NAEP). Although 
the YALS included a small group of items drawn from the NAEP reading 
assessment for linkage purposes, the two types of surveys differ sufficiently to 
m^e them noncomparable. That is, one can determine how well a young adult 
might do on the NAEP reading scale, based upon responses to the NAEP items 
in the YALS, but one cannot, from these data, equate scores on the NAEP 
reading assessment with scores on the YALS document, quantitative, or prose 
scales. This incompatibility between the NAEP reading assessment and the 
YALS (and NALS) impedes longitudinal analyses of functional literacy 
abilities. For example, reading scores from the NAEP high school reading tests 
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cannoi be compared to functional literacy scores from the same individuals, 
obtained in the years after leaving high school. 

Skills Assessments 

Most tests used for pre- and posttesting of adult skill abilities in lite. .'.cy 
programs have been basic skills tests, most often developed from similar 
tests for children. A typical adult reading test, for example, will test various 
types of comprehension (e.g., literal, inferential, and critical), plus 
vocabulary and perhaps structural analysis (e.g., prefixes and suffixes). 
Many of these tests report scores in grade-level equivalents, although other 
reporting procedures are possible. Grade-level equivalents are questionable 
scalings for elementary- and secondary-level assessment; for adults they are 
even more problematic (Kirsch & Guthrie, 1978; Ryan & Furlong, 1975; 
Sticht, 1988). A sixth grader who reads at a sixth-grade level and an adult 
who reads at a sixth-grade level usually have widely different reading 
abilities and require different forms of instruction, yet by grade-level 
measures they are classed as identical. In addition, grade-level designations 
for adults are misleading in that they assume the average experiential and 
background knowledge levels of children at the designated grade points. 
Although no widely accepted alternative scaling for adult abilities has yet 
been developed, the need for such a replacement is great. 

If functional literacy tests were simply alternative means for tapping the 
same skills as those assessed by basic skills tests, the two could be equated. 
However, for adults in ABE programs, the correlation between the document 
scale of the Test of Adult Literacy Skills (TALS; Simon & Schuster, 1990) 
and the reading comprehension scale of the Tests of Adult Basic Education 
(TABE; CTB/McGraw-Hill, 1987) is less than .70. A regression analysis 
found that the best predictor of document scale performance on the TALS 
was the TABE Mathematical Concepts and Applications; the TABE Reading 
Comprehension scores also made a significant but far smaller contribution to 
the predicted scores (Sabatini, Venezky, & Bristow, 1995). Functional 
literacy tests like the TALS require not only general reading comprehension, 
but also problem-solving abilities. 

The incompatibility between basic skills tests and national literacy 
surveys places a special burden on policymakers. The NALS results are not 
compatible with scores reported by literacy programs that use basic skills 
tests, and, while NALS scores could be converted to a crude grade-level 
equivalency, this would have little validity in that most of the NALS tasks are 
not taught extensively in school. Furthermore, the scores would still not be 
any more comparable than math and reading scores that were mapped onto 
grade-level scsdes. 

PROGRAM Evaluation 

One reason for administering formal tests at the beginning of an 
instructional program is to meet reporting requirements for student progress 
and program outcomes. Typically, basic skills gain scores are reported. 
There are many problems with this, and the results, as currently reported, 
have relatively limited use for policymakers. First, the pretest scores are 
based on a test that is generally administered within the first week that an 
adult has enrolled in a program. For many adults, this is the first encounter 
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with formal testing since dropping out of high school (or grade school). Scores 
tend, therefore, to be artificially low: Adults lack test-taking skills, have low 
self-confidence, and lack practice with some literacy skills. After even a few 
weeks in class, scores may rise significantly due simply to exposure to test 
formats, printed vocabulary, and the like. In the data cited above from New 
York City literacy programs, it is doubtful that the 9.9 months average gain 
reported for the first year of instruction represents a true increase in reading 
ability as opposed to a mixture of improved reading ability and improved test- 
taking strategies. 

Second, almost all test score reporting is fcr reading comprehension, yet 
most ABE courses stress a variety of skills, including basic mathematics, 
reading vocabulary and comprehension, and writing, as well as specific life 
skills (applying for a job, interacting with your child’s school, etc.). This 
means an incomplete match betw'een course content and test content. Part of the 
problem with current practices is that ABE programs are attempting to serve two 
distinct goals. One is to prepare adults who have not had the advantages of 
formal education or did not make sufficient progress in a formal educational 
system to cope effectively in a print-based society. This path might lead to GED 
certification and, for some, to community college or a four-year institution. The 
second goal is to give adults the practical skills they need for self-esteem, work, 
home, and civic needs. In this path, less attention is given to academic skills, 
which means less time for theories and other abstractions. (For a discussion of 
functional vs. general or academic literacy instruction, see Sticht, 1987). 

For evaluation of progress, several procedures should be considered. For 
academic programs, it is important to know how many students eventually 
reach their academic goals (GED certification, college, etc.) and in how much 
time. This information would allow policymakers to determine the overall 
effectiveness of the system as well as the relative effectiveness of different 
programs in helping students reach their academic goals. The GED examination 
and college entry requirements generally serve as checks on actual skill 
attainment, although with open admissions, it may also be important to track 
progress within post-secondary education. 

Students enrolled in courses that stress practical skills could be assessed at 
the termination of their instruction with functional literacy tests. To measure 
actual progress, poattest scores might be compared to scores obtained after a 
month or so of enrollment. But we have little agreement on what average gains 
to expect for different entry-level abilities and instructional regimens, and gain 
scores do not translate easily into policy. A gain of 50 points on the TALS 
document scale, for example, might be a positive sign for someone who entered 
a program with low reading ability and re-enrolled for the next higher level of 
instruction. For a person who entered with the same low level of ability and 
planned not to proceed to the next level, this may not be a positive sign, 
particularly if the exit level of functioning were .still too low to ensure adequate 
handling of print material for home, work, and civic life. 

An alternative is to report a combination of exit scores and enrollment 
statistics, including the percentage of students who enroll in the next higher 
level course, when one is available. This approach requires further work, 
however, on matching functional literacy scores with levels of functioning in 
society. What policymakers need to know, for those who enroll in literacy 
courses, is how many stay long enough to reach desired levels of functioning. 
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At present, many students who enroll do so on a casual basis, attending 
sporadically and often leaving without formally withdrawing. 

Another complication to consider is the curriculum-assessment mismatch 
discussed above. For efficiency, we would like to have assessment 
instruments that could inform instruction as well as policy. However, this 
may not be possible. Intelligent policy requires data comparable across time 
and geographic region. Furthermore, policy-related results must be 
interpretable in relation to societal goals. Students who enroll in literacy 
courses to learn how to read to their children may not care at the end of 
instruction what their reading levels are relative to national standards. 
However, policymakers do. One option is to divide students according to 
their expressed goals into general literacy students and specialized skill (or 
functional literacy) students. Then, only the former would be tested with 
nationally standardized instruments. The latter group might not be assessed at 
all or be assessed through alternative means. A second option is to require 
that all students be tested on exit with nationally standardized instruments as 
a condition of enrollment. This by itself, however, will not solve the problem 
created by those who drop out before exit testing. 

Score Reporting 

As stated earlier, grade-level reporting for test scores needs to be 
reconsidered. If it is an adequate scale for reporting adult abilities, then it 
needs to be justified both empirically and logically. If not, then a scale (or 
scales) that applies more directly to adult functioning levels needs to be 
developed. Such a scale might be developed around materials and tasks that 
represent different levels of functioning in work, home, and civic life. 
Alternatively, the scale might be built around knowledge of cognitive 
processing, the skills and strategies required for different levels of ability. A 
sufficient base already exists for initiating work on the latter. Studies of word 
identification reaction times, decoding, reading flexibility, and 
metacognition, in particular, provide insights into some of the cognitive skills 
required for reading (e.g., Perfetti, 1985; Stanovich & West, 1989; Vellutino 
& Scanlon, 1987). A targeted research program could pursue these and other 
leads with adults, not just for reading but also for writing. 

The differences between these two approaches to the development of an 
alternative literacy scale for adults should subjected to critical examination. 
The materials and tasks approach will be easier to relate to home, work, and 
civic ftinctioning; the cognitive processing approach will be easier to relate to 
elementary and secondary assessment and to instruction. The former will 
need to be re-evaluated as the literacy demands of everyday life change, 
while the latter will require an additional step to relate it to everyday literacy 
demands. 
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Toward a More Focused 
Database 



Perhaps the most important lesson to be learned from national literacy 
surveys is that they have limitations for policy formation and evaluation. This is 
not to invalidate what are clearly important contributions toward an 
understanding of the distribution of literacy skills in a population, but rather to 
recognize the complexity of the problem that is faced. What is needed in most 
developed countries is not fiirther information on literacy in the total population, 
but more focused information on the groups who need assistance and the 
effectiveness of programs instituted to assist them. The continued practice of 
aggregating disparate needs under the literacy banner disguises the vast 
differences across the individuals and groups involved. 

The settlement and integration of immigrants and refugees, for example, 
potentially requires language and cultural instruction, along with skill training 
for jobs. Policy evaluation for this population must attend to the initial support 
system provided, the effectiveness of second language and culture instruction, 
the matching of skill training to available jobs; and the progress of families 
toward economic independence, educational advancement, and citizenship. 

For the native-born with low literacy and job skills, different policies are 
needed, oriented toward the specific life situations of the individuals involved. 
Policy for young mothers on welfare who are high school dropouts, for 
example, might be focused on completion of high school since modeling of 
educational alternatives shows distinct long-term economic advantages for this 
alternative over GED certification. For policy evaluation, data would then be 
needed on program types, enrollments, educational progress, and longer term 
job attainment and earnings. Similar data are needed on inmates of correctional 
institutions, with recidivism monitoring, and on other groups, with 
consideration of their special needs. 

Evaluation of program effectiveness is probably the most serious issue 
faced by literacy policymakers, yet limited scholarly attention is devoted to this 
issue. What is usually discussed, furthermore, is evaluation of particular 
programs instead of evaluation of policy options. More attention needs to be 
devoted to modeling of policy options, with assessment and evaluation 
designed to validate the accuracy of predictions. If policy options became the 
focus of literacy evaluation, then a serious flaw in the present system would 
become obvious, this being the lack of connectivity between literacy programs 
and policy objectives. At present, the United States has a number of disjointed 
systems for assisting the poor, indigent, immigrants, and the like. Each system 
or set of programs has an ostensible goal of assisting people to become 
independent citizens, economically self-sustaining. The literacy component, 
however, is seldom viewed in terms of its contribution to this goal. Instead, it 
has become an end unto itself. 

Arbitrary gains in literacy ability are sometimes set as objectives for 
programs, without regard for either what is reasonable to expect for the student 
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population involved or necessary for the goals of the program. Data 
collection for adult literacy has taken on a similar static and disjointed 
approach. We collect data on numbers of people in programs, on literacy 
abilities in the population, on average length of attendance in programs, and 
more, yet we seldom question what these data mean about literacy policy , 
and we rarely motivate data collection by specific policy issues. Most data 
collected on adult literacy are justified by a general academic interest in 
knowing, not by a specific need for policy evaluation. 

To construct a better knowledge base for literacy policy forrnation and 
evaluation, we must first clarify policy objectives in relation to individual 
outcomes. How, for example, is ABE, ESL, and ASE funding to lead to 
economic independence for specific populations? Where do welfare support 
and job training fit into the total system? What are our expectations for the 
yearly progress of specific subpopulations like high school dropouts? How 
is the system coupled to local and regional job markets? How do we ensure 
that individuals are being prepared for jobs that will exist when they are 
ready to enter the job market? Many technical and vocational colleges have 
advisory boards for trades so that current information on skill needs and job 
potential is incorporated into admissions and curriculum. A similar 
relationship between occupations and training is needed for literacy 
programs. 

Once we decide on the most likely routes for moving individuals with 
low skills and limited economic potential to economic independence, we 
must then design data-gathering for monitoring the progress of people who 
move along each path. Then, we should no longer ask whether or not 
participants in ABE programs are making a grade-level of progress for each 
year enrolled or what the document literacy is of high school graduates. 
Instead, we should ask what percentage of the individuals who began 
training in a particular year progressed as expected toward employment, 
whether the skills taught in particular literacy programs matched those needed 
for local jobs and for social and family needs, and whether the supply of 
literacy programs matched the demand within a community or region. As 
long as our only goal is to raise literacy levels independent of specific needs, 
acsessment will serve mainly to feed vast storehouses of data that will lie in 
cluttered obscurity around the countryside like decaying old gas stations 
along roads no longer taken, justified by a solitary burst of publicity at 
release time and forgotten soon after. 
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